Skip to content

feat(s3): add a blob-backed S3-compatible API#53

Draft
pthmas wants to merge 9 commits intomainfrom
pthmas/s3-blob-export
Draft

feat(s3): add a blob-backed S3-compatible API#53
pthmas wants to merge 9 commits intomainfrom
pthmas/s3-blob-export

Conversation

@pthmas
Copy link
Copy Markdown
Collaborator

@pthmas pthmas commented Apr 28, 2026

Summary

  • add a SQLite-backed S3-compatible API for bucket and object CRUD
  • route S3 uploads through the shared Celestia submission backend and persist blob metadata
  • extend the e2e harness and CI wiring to cover indexing, gRPC, and S3 flows

@pthmas pthmas self-assigned this Apr 28, 2026
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 28, 2026

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 8db7f772-c128-44ce-b5fb-359bc87b0b61

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch pthmas/s3-blob-export

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@pthmas pthmas changed the title feat(s3): add e2e coverage for blob-backed API feat(s3): add a blob-backed S3-compatible API Apr 29, 2026
pthmas and others added 6 commits April 29, 2026 10:59
Write operations (CreateBucket, DeleteBucket, PutObject, DeleteObject)
now return ErrReadOnly immediately when no submitter is wired in.
Previously PutObject would silently write to SQLite with no Celestia
anchor (height=0, empty commitments), producing orphaned data.

ErrReadOnly maps to 405 MethodNotAllowed in the HTTP error handler.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- object.go: escape LIKE wildcards (%, _, \) in ListObjects prefix to
  prevent unintended pattern matching on user-supplied keys
- object.go: populate Object.Namespace from o.ns on read instead of
  scanning per-row DB value, eliminating stale-config drift
- server.go: remove dead handleBucket wrapper, route GET bucket
  directly to handleListObjects
- server.go: clamp max-keys to 1000 per S3 spec
- server.go: apply http.MaxBytesReader before reading PUT body so
  oversized requests are rejected at the network layer, not after
  buffering the full payload in memory
- service.go: detect http.MaxBytesError from MaxBytesReader and map
  to ErrObjectTooLarge
- auth.go: validate X-Amz-Date is within ±15 min to prevent replay
  attacks with captured signed requests
- tests: update to use mockSubmitter for write ops, fix stale
  hardcoded SigV4 timestamp, add TestService_ReadOnly coverage

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Empty PUTs skip Celestia submission and store locally with Height=0
and no commitments. This is intentional to preserve S3 tool
compatibility (e.g. folder placeholder keys like "prefix/").

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Critical fixes:
- server.go: parsePath now uses r.URL.RawPath to avoid double-decoding;
  percent-encoded characters in keys (e.g. %2F) are preserved through
  path splitting and decoded per-segment
- server.go: remove query-param priority over HTTP method in bucket
  router; DELETE/PUT/HEAD on a bucket with query params now routes
  correctly instead of falling through to handleListObjects
- object.go: wrap DeleteBucket count check and delete in a single
  transaction to close TOCTOU race where a concurrent write could sneak
  in between the two separate queries

Medium fixes:
- object.go: remove redundant GetBucket call from PutObject; the SQLite
  FK constraint enforces bucket existence and is detected via
  isSQLiteFKConstraint → ErrBucketNotFound
- migrations/005: drop idx_s3_objects_bucket; the composite index on
  (bucket, key) already covers all bucket-only lookups
- service.go: validate bucket names (3-63 chars, lowercase alphanum +
  hyphen, no leading/trailing hyphen, not an IP address) and key length
  (max 1024 bytes); new ErrInvalidBucketName and ErrKeyTooLong errors
  map to 400 in the HTTP layer

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant